{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson2 表格型方法——Sarsa\n",
    "## 1. Sarsa 简介\n",
    "* `Sarsa`全称是`state-action-reward-state'-action'`，目的是学习特定的`state`下，特定`action`的价值`Q`，最终建立和优化一个`Q`表格，以`state`为行，`action`为列，根据与环境交互得到的`reward`来更新`Q`表格，更新公式为：\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/776b473b7f994702a3e05c5eac1156a7ce03b9e6bdb5453085fa9cbb86979715)\n",
    "* `Sarsa`在训练中为了更好的探索环境，采用`ε-greedy`方式来训练，有一定概率随机选择动作输出。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from q_table.Games.Games import SnakeEnv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agent\n",
    "* `Agent`是和环境`environment`交互的主体。\n",
    "* `predict()`方法：输入观察值`observation`（或者说状态`state`），输出动作值\n",
    "* `sample()`方法：再`predict()`方法基础上使用`ε-greedy`增加探索\n",
    "* `learn()`方法：输入训练数据，完成一轮`Q`表格的更新\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "class SarsaAgent(object):\n",
    "    def __init__(self, obs_n, act_n, learning_rate=0.01, gamma=0.9, e_greed=0.1):\n",
    "        self.act_n = act_n  # 动作维度，有几个动作可选\n",
    "        self.lr = learning_rate  # 学习率\n",
    "        self.gamma = gamma  # reward的衰减率\n",
    "        self.epsilon = e_greed  # 按一定概率随机选动作\n",
    "        self.Q = np.zeros((obs_n, act_n))\n",
    "\n",
    "    # 根据输入观察值，采样输出的动作值，带探索\n",
    "    def sample(self, obs):\n",
    "        if np.random.uniform(0, 1) < (1.0 - self.epsilon):  #根据table的Q值选动作\n",
    "            action = self.predict(obs)\n",
    "        else:\n",
    "            action = np.random.choice(self.act_n)  #有一定概率随机探索选取一个动作\n",
    "        return action\n",
    "\n",
    "    # 根据输入观察值，预测输出的动作值\n",
    "    def predict(self, obs):\n",
    "        Q_list = self.Q[obs, :]\n",
    "        maxQ = np.max(Q_list)\n",
    "        action_list = np.where(Q_list == maxQ)[0]  # maxQ可能对应多个action\n",
    "        action = np.random.choice(action_list)\n",
    "        return action\n",
    "\n",
    "    # 学习方法，也就是更新Q-table的方法\n",
    "    def learn(self, obs, action, reward, next_obs, next_action, done):\n",
    "        \"\"\" on-policy\n",
    "            obs: 交互前的obs, s_t\n",
    "            action: 本次交互选择的action, a_t\n",
    "            reward: 本次动作获得的奖励r\n",
    "            next_obs: 本次交互后的obs, s_t+1\n",
    "            next_action: 根据当前Q表格, 针对next_obs会选择的动作, a_t+1\n",
    "            done: episode是否结束\n",
    "            Q(S<t>,A<t>)=Q(S<t>,A<t>) + α[R<t+1>+γQ(S<t+1>,A<t+1>) - Q(S<t>,A<t>)]\n",
    "            Q(S<t>,A<t>) == 在t时刻,状态为S<t>, 采取A<t>行动的Q值\n",
    "            R<t> == 在t-1时刻采取行动之后的奖励\n",
    "            α == 学习率\n",
    "            γ == 衰减率\n",
    "        \"\"\"\n",
    "        predict_Q = self.Q[obs, action]\n",
    "        if done:\n",
    "            target_Q = reward  # 没有下一个状态了\n",
    "        else:\n",
    "            target_Q = reward + self.gamma * self.Q[next_obs, next_action]  # Sarsa 对应 R<t+1>+γQ(S<t+1>,A<t+1>)\n",
    "        self.Q[obs, action] += self.lr * (target_Q - predict_Q)  # 修正q\n",
    "\n",
    "    # 保存Q表格数据到文件\n",
    "    def save(self):\n",
    "        npy_file = './q_table.npy'\n",
    "        np.save(npy_file, self.Q)\n",
    "        print(npy_file + ' saved.')\n",
    "\n",
    "    # 从文件中读取Q值到Q表格中\n",
    "    def restore(self, npy_file='./q_table.npy'):\n",
    "        self.Q = np.load(npy_file)\n",
    "        print(npy_file + ' loaded.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Training && Test（训练&&测试）\n",
    "* `run_episode()`：`agent`在一个`episode`中训练的过程，使用`agent.sample()`与环境交互，使用`agent.learn()`训练`Q`表格。\n",
    "* `test_episode()`：`agent`在一个`episode`中测试效果，评估目前的`agent`能在一个`episode`中拿到多少总`reward`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Training:\n",
    "    @staticmethod\n",
    "    def run_episode(env, agent, render=False):\n",
    "        total_steps = 0  # 记录每个episode走了多少step\n",
    "        total_reward = 0\n",
    "\n",
    "        obs = env.reset()  # 重置环境, 重新开一局（即开始新的一个episode）\n",
    "        action = agent.sample(obs)  # 根据算法选择一个动作\n",
    "\n",
    "        while True:\n",
    "            next_obs, reward, done, _ = env.step(action)  # 与环境进行一个交互\n",
    "            next_action = agent.sample(next_obs)  # 根据算法选择一个动作\n",
    "            # 训练 Sarsa 算法\n",
    "            agent.learn(obs, action, reward, next_obs, next_action, done)\n",
    "\n",
    "            action = next_action\n",
    "            obs = next_obs  # 存储上一个观察值\n",
    "            total_reward += reward\n",
    "            total_steps += 1  # 计算step数\n",
    "            if render:\n",
    "                env.render()  #渲染新的一帧图形\n",
    "            if done:\n",
    "                break\n",
    "        return total_reward, total_steps\n",
    "\n",
    "    @staticmethod\n",
    "    def test_episode(env, agent):\n",
    "        total_reward = 0\n",
    "        obs = env.reset()\n",
    "\n",
    "        while True:\n",
    "            action = agent.predict(obs)  # greedy\n",
    "            next_obs, reward, done, _ = env.step(action)\n",
    "            total_reward += reward\n",
    "            obs = next_obs\n",
    "            # time.sleep(0.5)\n",
    "            # env.render()\n",
    "            if done:\n",
    "                break\n",
    "        return total_reward\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建环境和Agent，启动训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ladders info:{80: 86, 78: 25, 55: 64, 37: 8, 27: 67, 50: 24, 12: 73, 60: 38, 85: 92, 66: 62, 87: 68, 51: 3, 97: 76, 33: 28, 18: 83, 35: 57, 1: 72, 96: 91, 70: 84, 32: 14, 86: 80, 25: 78, 64: 55, 8: 37, 67: 27, 24: 50, 73: 12, 38: 60, 92: 85, 62: 66, 68: 87, 3: 51, 76: 97, 28: 33, 83: 18, 57: 35, 72: 1, 91: 96, 84: 70, 14: 32} dice ranges:[0, 1]\n"
     ]
    }
   ],
   "source": [
    "env = SnakeEnv(20, [0, 1])\n",
    "env.ladders = {2: 46, 31: 19, 95: 51, 25: 17, 40: 23, 84: 26, 88: 30, 43: 85, 96: 13, 76: 12, 35: 36, 65: 79, 24: 68,\n",
    "               56: 94, 53: 77, 27: 60, 71: 45, 81: 6, 72: 32, 16: 69, 46: 2, 19: 31, 51: 95, 17: 25, 23: 40, 26: 84,\n",
    "               30: 88, 85: 43, 13: 96, 12: 76, 36: 35, 79: 65, 68: 24, 94: 56, 77: 53, 60: 27, 45: 71, 6: 81, 32: 72,\n",
    "               69: 16}\n",
    "agent = SarsaAgent(\n",
    "    obs_n=env.observation_space.n,\n",
    "    act_n=env.action_space.n,\n",
    "    learning_rate=0.1,\n",
    "    gamma=0.9,\n",
    "    e_greed=0.1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./q_table.npy loaded.\n",
      "Episode 0: steps=53, reward=48Episode 1: steps=176, reward=-75Episode 2: steps=79, reward=22Episode 3: steps=36, reward=65Episode 4: steps=111, reward=-10Episode 5: steps=18, reward=83Episode 6: steps=10, reward=91Episode 7: steps=139, reward=-38Episode 8: steps=7, reward=94Episode 9: steps=38, reward=63Episode 10: steps=48, reward=53Episode 11: steps=67, reward=34Episode 12: steps=32, reward=69Episode 13: steps=17, reward=84Episode 14: steps=39, reward=62Episode 15: steps=58, reward=43Episode 16: steps=77, reward=24Episode 17: steps=51, reward=50Episode 18: steps=7, reward=94Episode 19: steps=106, reward=-5Episode 20: steps=4, reward=97Episode 21: steps=78, reward=23Episode 22: steps=160, reward=-59Episode 23: steps=33, reward=68Episode 24: steps=22, reward=79Episode 25: steps=38, reward=63Episode 26: steps=27, reward=74Episode 27: steps=36, reward=65Episode 28: steps=43, reward=58Episode 29: steps=79, reward=22Episode 30: steps=2, reward=99Episode 31: steps=4, reward=97Episode 32: steps=29, reward=72Episode 33: steps=9, reward=92Episode 34: steps=19, reward=82Episode 35: steps=19, reward=82Episode 36: steps=153, reward=-52Episode 37: steps=37, reward=64Episode 38: steps=35, reward=66Episode 39: steps=12, reward=89Episode 40: steps=92, reward=9Episode 41: steps=61, reward=40Episode 42: steps=36, reward=65Episode 43: steps=10, reward=91Episode 44: steps=4, reward=97Episode 45: steps=148, reward=-47Episode 46: steps=22, reward=79Episode 47: steps=18, reward=83Episode 48: steps=87, reward=14Episode 49: steps=212, reward=-111Episode 50: steps=14, reward=87Episode 51: steps=93, reward=8Episode 52: steps=23, reward=78Episode 53: steps=9, reward=92Episode 54: steps=45, reward=56Episode 55: steps=66, reward=35Episode 56: steps=29, reward=72Episode 57: steps=21, reward=80Episode 58: steps=16, reward=85Episode 59: steps=86, reward=15Episode 60: steps=52, reward=49Episode 61: steps=79, reward=22Episode 62: steps=124, reward=-23Episode 63: steps=16, reward=85Episode 64: steps=3, reward=98Episode 65: steps=115, reward=-14Episode 66: steps=243, reward=-142Episode 67: steps=38, reward=63Episode 68: steps=22, reward=79Episode 69: steps=50, reward=51Episode 70: steps=6, reward=95Episode 71: steps=41, reward=60Episode 72: steps=149, reward=-48Episode 73: steps=61, reward=40Episode 74: steps=10, reward=91Episode 75: steps=16, reward=85Episode 76: steps=48, reward=53Episode 77: steps=52, reward=49Episode 78: steps=8, reward=93Episode 79: steps=68, reward=33Episode 80: steps=131, reward=-30Episode 81: steps=12, reward=89Episode 82: steps=45, reward=56Episode 83: steps=249, reward=-148Episode 84: steps=19, reward=82Episode 85: steps=91, reward=10Episode 86: steps=20, reward=81Episode 87: steps=11, reward=90Episode 88: steps=10, reward=91Episode 89: steps=87, reward=14Episode 90: steps=48, reward=53Episode 91: steps=28, reward=73Episode 92: steps=6, reward=95Episode 93: steps=39, reward=62Episode 94: steps=67, reward=34Episode 95: steps=42, reward=59Episode 96: steps=51, reward=50Episode 97: steps=36, reward=65Episode 98: steps=18, reward=83Episode 99: steps=20, reward=81Episode 100: steps=58, reward=43Episode 101: steps=49, reward=52Episode 102: steps=76, reward=25Episode 103: steps=64, reward=37Episode 104: steps=62, reward=39Episode 105: steps=56, reward=45Episode 106: steps=86, reward=15Episode 107: steps=181, reward=-80Episode 108: steps=32, reward=69Episode 109: steps=27, reward=74Episode 110: steps=21, reward=80Episode 111: steps=81, reward=20Episode 112: steps=87, reward=14Episode 113: steps=19, reward=82Episode 114: steps=43, reward=58Episode 115: steps=20, reward=81Episode 116: steps=117, reward=-16Episode 117: steps=59, reward=42Episode 118: steps=7, reward=94Episode 119: steps=107, reward=-6Episode 120: steps=43, reward=58Episode 121: steps=98, reward=3Episode 122: steps=57, reward=44Episode 123: steps=107, reward=-6Episode 124: steps=41, reward=60Episode 125: steps=28, reward=73Episode 126: steps=9, reward=92Episode 127: steps=128, reward=-27Episode 128: steps=22, reward=79Episode 129: steps=14, reward=87Episode 130: steps=50, reward=51Episode 131: steps=55, reward=46Episode 132: steps=101, reward=0Episode 133: steps=50, reward=51Episode 134: steps=8, reward=93Episode 135: steps=86, reward=15Episode 136: steps=41, reward=60Episode 137: steps=26, reward=75Episode 138: steps=62, reward=39Episode 139: steps=84, reward=17Episode 140: steps=227, reward=-126Episode 141: steps=166, reward=-65Episode 142: steps=22, reward=79Episode 143: steps=7, reward=94Episode 144: steps=273, reward=-172Episode 145: steps=60, reward=41Episode 146: steps=22, reward=79Episode 147: steps=15, reward=86Episode 148: steps=28, reward=73Episode 149: steps=16, reward=85Episode 150: steps=47, reward=54Episode 151: steps=44, reward=57Episode 152: steps=58, reward=43Episode 153: steps=43, reward=58Episode 154: steps=43, reward=58Episode 155: steps=99, reward=2Episode 156: steps=40, reward=61Episode 157: steps=43, reward=58Episode 158: steps=94, reward=7Episode 159: steps=8, reward=93Episode 160: steps=111, reward=-10Episode 161: steps=43, reward=58Episode 162: steps=107, reward=-6Episode 163: steps=117, reward=-16Episode 164: steps=131, reward=-30Episode 165: steps=76, reward=25Episode 166: steps=67, reward=34Episode 167: steps=85, reward=16Episode 168: steps=28, reward=73Episode 169: steps=26, reward=75Episode 170: steps=12, reward=89Episode 171: steps=37, reward=64Episode 172: steps=40, reward=61Episode 173: steps=73, reward=28Episode 174: steps=54, reward=47Episode 175: steps=14, reward=87Episode 176: steps=101, reward=0Episode 177: steps=151, reward=-50Episode 178: steps=17, reward=84Episode 179: steps=28, reward=73Episode 180: steps=27, reward=74Episode 181: steps=109, reward=-8Episode 182: steps=31, reward=70Episode 183: steps=142, reward=-41Episode 184: steps=27, reward=74Episode 185: steps=15, reward=86Episode 186: steps=53, reward=48Episode 187: steps=73, reward=28Episode 188: steps=95, reward=6Episode 189: steps=83, reward=18Episode 190: steps=46, reward=55Episode 191: steps=13, reward=88Episode 192: steps=164, reward=-63Episode 193: steps=49, reward=52Episode 194: steps=106, reward=-5Episode 195: steps=93, reward=8Episode 196: steps=21, reward=80Episode 197: steps=23, reward=78Episode 198: steps=44, reward=57Episode 199: steps=20, reward=81Episode 200: steps=60, reward=41Episode 201: steps=91, reward=10Episode 202: steps=84, reward=17Episode 203: steps=26, reward=75Episode 204: steps=46, reward=55Episode 205: steps=20, reward=81Episode 206: steps=61, reward=40Episode 207: steps=23, reward=78Episode 208: steps=17, reward=84Episode 209: steps=10, reward=91Episode 210: steps=23, reward=78Episode 211: steps=24, reward=77Episode 212: steps=39, reward=62Episode 213: steps=14, reward=87Episode 214: steps=28, reward=73Episode 215: steps=292, reward=-191Episode 216: steps=54, reward=47Episode 217: steps=13, reward=88Episode 218: steps=59, reward=42Episode 219: steps=24, reward=77Episode 220: steps=34, reward=67Episode 221: steps=28, reward=73Episode 222: steps=4, reward=97Episode 223: steps=97, reward=4Episode 224: steps=128, reward=-27Episode 225: steps=98, reward=3Episode 226: steps=103, reward=-2Episode 227: steps=16, reward=85Episode 228: steps=83, reward=18Episode 229: steps=36, reward=65Episode 230: steps=18, reward=83Episode 231: steps=13, reward=88Episode 232: steps=5, reward=96Episode 233: steps=44, reward=57Episode 234: steps=7, reward=94Episode 235: steps=15, reward=86Episode 236: steps=94, reward=7Episode 237: steps=75, reward=26Episode 238: steps=66, reward=35Episode 239: steps=6, reward=95Episode 240: steps=15, reward=86Episode 241: steps=9, reward=92Episode 242: steps=36, reward=65Episode 243: steps=49, reward=52Episode 244: steps=22, reward=79Episode 245: steps=10, reward=91Episode 246: steps=89, reward=12Episode 247: steps=11, reward=90Episode 248: steps=65, reward=36Episode 249: steps=44, reward=57Episode 250: steps=2, reward=99Episode 251: steps=56, reward=45Episode 252: steps=44, reward=57Episode 253: steps=8, reward=93Episode 254: steps=4, reward=97Episode 255: steps=36, reward=65Episode 256: steps=19, reward=82Episode 257: steps=11, reward=90Episode 258: steps=44, reward=57Episode 259: steps=25, reward=76Episode 260: steps=3, reward=98Episode 261: steps=21, reward=80Episode 262: steps=79, reward=22Episode 263: steps=21, reward=80Episode 264: steps=8, reward=93Episode 265: steps=20, reward=81Episode 266: steps=13, reward=88Episode 267: steps=7, reward=94Episode 268: steps=4, reward=97Episode 269: steps=23, reward=78Episode 270: steps=149, reward=-48Episode 271: steps=21, reward=80Episode 272: steps=4, reward=97Episode 273: steps=21, reward=80Episode 274: steps=38, reward=63Episode 275: steps=23, reward=78Episode 276: steps=89, reward=12Episode 277: steps=79, reward=22Episode 278: steps=14, reward=87Episode 279: steps=59, reward=42Episode 280: steps=5, reward=96Episode 281: steps=28, reward=73Episode 282: steps=24, reward=77Episode 283: steps=41, reward=60Episode 284: steps=17, reward=84Episode 285: steps=14, reward=87Episode 286: steps=16, reward=85Episode 287: steps=42, reward=59Episode 288: steps=40, reward=61Episode 289: steps=24, reward=77Episode 290: steps=23, reward=78Episode 291: steps=77, reward=24Episode 292: steps=81, reward=20Episode 293: steps=35, reward=66Episode 294: steps=155, reward=-54Episode 295: steps=11, reward=90Episode 296: steps=51, reward=50Episode 297: steps=68, reward=33Episode 298: steps=184, reward=-83Episode 299: steps=46, reward=55Episode 300: steps=127, reward=-26Episode 301: steps=8, reward=93Episode 302: steps=286, reward=-185Episode 303: steps=10, reward=91Episode 304: steps=29, reward=72Episode 305: steps=70, reward=31Episode 306: steps=136, reward=-35Episode 307: steps=76, reward=25Episode 308: steps=50, reward=51Episode 309: steps=48, reward=53Episode 310: steps=111, reward=-10Episode 311: steps=16, reward=85Episode 312: steps=42, reward=59Episode 313: steps=7, reward=94Episode 314: steps=42, reward=59Episode 315: steps=27, reward=74Episode 316: steps=5, reward=96Episode 317: steps=119, reward=-18Episode 318: steps=22, reward=79Episode 319: steps=23, reward=78Episode 320: steps=40, reward=61Episode 321: steps=132, reward=-31Episode 322: steps=38, reward=63Episode 323: steps=156, reward=-55Episode 324: steps=75, reward=26Episode 325: steps=100, reward=1Episode 326: steps=21, reward=80Episode 327: steps=27, reward=74Episode 328: steps=31, reward=70Episode 329: steps=2, reward=99Episode 330: steps=4, reward=97Episode 331: steps=46, reward=55Episode 332: steps=109, reward=-8Episode 333: steps=20, reward=81Episode 334: steps=24, reward=77Episode 335: steps=23, reward=78Episode 336: steps=23, reward=78Episode 337: steps=10, reward=91Episode 338: steps=20, reward=81Episode 339: steps=31, reward=70Episode 340: steps=102, reward=-1Episode 341: steps=152, reward=-51Episode 342: steps=15, reward=86Episode 343: steps=10, reward=91Episode 344: steps=41, reward=60Episode 345: steps=24, reward=77Episode 346: steps=27, reward=74Episode 347: steps=4, reward=97Episode 348: steps=39, reward=62Episode 349: steps=52, reward=49Episode 350: steps=77, reward=24Episode 351: steps=41, reward=60Episode 352: steps=146, reward=-45Episode 353: steps=87, reward=14Episode 354: steps=31, reward=70Episode 355: steps=70, reward=31Episode 356: steps=54, reward=47Episode 357: steps=96, reward=5Episode 358: steps=4, reward=97Episode 359: steps=57, reward=44Episode 360: steps=8, reward=93Episode 361: steps=6, reward=95Episode 362: steps=69, reward=32Episode 363: steps=24, reward=77Episode 364: steps=41, reward=60Episode 365: steps=13, reward=88Episode 366: steps=16, reward=85Episode 367: steps=59, reward=42Episode 368: steps=53, reward=48Episode 369: steps=109, reward=-8Episode 370: steps=36, reward=65Episode 371: steps=39, reward=62Episode 372: steps=61, reward=40Episode 373: steps=134, reward=-33Episode 374: steps=35, reward=66Episode 375: steps=140, reward=-39Episode 376: steps=115, reward=-14Episode 377: steps=62, reward=39Episode 378: steps=15, reward=86Episode 379: steps=94, reward=7Episode 380: steps=264, reward=-163Episode 381: steps=86, reward=15Episode 382: steps=13, reward=88Episode 383: steps=35, reward=66Episode 384: steps=36, reward=65Episode 385: steps=44, reward=57Episode 386: steps=45, reward=56Episode 387: steps=152, reward=-51Episode 388: steps=39, reward=62Episode 389: steps=21, reward=80Episode 390: steps=237, reward=-136Episode 391: steps=40, reward=61Episode 392: steps=6, reward=95Episode 393: steps=11, reward=90Episode 394: steps=9, reward=92Episode 395: steps=47, reward=54Episode 396: steps=5, reward=96Episode 397: steps=28, reward=73Episode 398: steps=207, reward=-106Episode 399: steps=27, reward=74Episode 400: steps=59, reward=42Episode 401: steps=25, reward=76Episode 402: steps=25, reward=76Episode 403: steps=14, reward=87Episode 404: steps=23, reward=78Episode 405: steps=21, reward=80Episode 406: steps=311, reward=-210Episode 407: steps=14, reward=87Episode 408: steps=191, reward=-90Episode 409: steps=60, reward=41Episode 410: steps=64, reward=37Episode 411: steps=20, reward=81Episode 412: steps=28, reward=73Episode 413: steps=49, reward=52Episode 414: steps=87, reward=14Episode 415: steps=60, reward=41Episode 416: steps=14, reward=87Episode 417: steps=192, reward=-91Episode 418: steps=85, reward=16Episode 419: steps=75, reward=26Episode 420: steps=15, reward=86Episode 421: steps=5, reward=96Episode 422: steps=144, reward=-43Episode 423: steps=151, reward=-50Episode 424: steps=32, reward=69Episode 425: steps=10, reward=91Episode 426: steps=247, reward=-146Episode 427: steps=112, reward=-11Episode 428: steps=17, reward=84Episode 429: steps=19, reward=82Episode 430: steps=69, reward=32Episode 431: steps=57, reward=44Episode 432: steps=26, reward=75Episode 433: steps=8, reward=93Episode 434: steps=7, reward=94Episode 435: steps=13, reward=88Episode 436: steps=55, reward=46Episode 437: steps=25, reward=76Episode 438: steps=15, reward=86Episode 439: steps=7, reward=94Episode 440: steps=15, reward=86Episode 441: steps=11, reward=90Episode 442: steps=63, reward=38Episode 443: steps=21, reward=80Episode 444: steps=31, reward=70Episode 445: steps=26, reward=75Episode 446: steps=57, reward=44Episode 447: steps=41, reward=60Episode 448: steps=91, reward=10Episode 449: steps=168, reward=-67Episode 450: steps=18, reward=83Episode 451: steps=50, reward=51Episode 452: steps=159, reward=-58Episode 453: steps=23, reward=78Episode 454: steps=52, reward=49Episode 455: steps=112, reward=-11Episode 456: steps=242, reward=-141Episode 457: steps=14, reward=87Episode 458: steps=21, reward=80Episode 459: steps=27, reward=74Episode 460: steps=10, reward=91Episode 461: steps=124, reward=-23Episode 462: steps=53, reward=48Episode 463: steps=39, reward=62Episode 464: steps=139, reward=-38Episode 465: steps=10, reward=91Episode 466: steps=66, reward=35Episode 467: steps=21, reward=80Episode 468: steps=79, reward=22Episode 469: steps=204, reward=-103Episode 470: steps=22, reward=79Episode 471: steps=133, reward=-32Episode 472: steps=109, reward=-8Episode 473: steps=29, reward=72Episode 474: steps=44, reward=57Episode 475: steps=15, reward=86Episode 476: steps=11, reward=90Episode 477: steps=79, reward=22Episode 478: steps=35, reward=66Episode 479: steps=41, reward=60Episode 480: steps=30, reward=71Episode 481: steps=114, reward=-13Episode 482: steps=46, reward=55Episode 483: steps=42, reward=59Episode 484: steps=90, reward=11Episode 485: steps=41, reward=60Episode 486: steps=11, reward=90Episode 487: steps=75, reward=26Episode 488: steps=124, reward=-23Episode 489: steps=94, reward=7Episode 490: steps=78, reward=23Episode 491: steps=30, reward=71Episode 492: steps=46, reward=55Episode 493: steps=55, reward=46Episode 494: steps=4, reward=97Episode 495: steps=16, reward=85Episode 496: steps=25, reward=76Episode 497: steps=16, reward=85Episode 498: steps=21, reward=80Episode 499: steps=49, reward=52./q_table.npy saved.\n"
     ]
    }
   ],
   "source": [
    "# 训练500个episode，打印每个episode的分数\n",
    "agent.restore('./q_table.npy')\n",
    "for episode in range(500):\n",
    "    ep_reward, ep_steps = Training.run_episode(env, agent, False)\n",
    "    print(f'Episode {episode}: steps={ep_steps}, reward={ep_reward}', end='')\n",
    "agent.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test reward = 85.0\n"
     ]
    }
   ],
   "source": [
    "# 全部训练结束，查看算法效果\n",
    "test_reward = Training.test_episode(env, agent)\n",
    "print('test reward = %.1f' % (test_reward))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}